Video game system with wireless modular handheld controller

ABSTRACT

A home entertainment system for video games and other applications includes a main unit and handheld controllers. The handheld controllers sense their own motion by detecting illumination emitted by emitters positioned at either side of a display. The controllers can be plugged into expansion units that customize the overall control interface for particular applications including but not limited to legacy video games.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.11/532,328, filed Sep. 15, 2006, which claims priority from provisionalapplication No. 60/716,937, filed on Sep. 15, 2005. This application isalso related to U.S. application Ser. No. 11/446,187, filed on Jun. 5,2006; and U.S. application Ser. No. 11/446,188, filed on Jun. 5, 2006,the disclosures of which are incorporated herein by reference.

FIELD

The technology herein relates to consumer electronics, and moreparticularly to video game and entertainment systems. In still moredetail, the technology herein relates to a home video game systemincluding a modular remote wireless handheld control device withcapabilities including position sensing.

BACKGROUND AND SUMMARY

Computer graphics technology has come a long way since video games werefirst developed. Relatively inexpensive 3D graphics engines now providenearly photo-realistic interactive game play on home video game andpersonal computer hardware platforms costing only a few hundred dollars.

Most game players demand great graphics, but the core of video game playis the man-machine interface—the interaction between the (human) gameplayer and the gaming platform. Video games are fun and exciting to playbecause the game player can interact with the game and affect or controlthe gaming events and outcome. Since the essence of an enjoyable videogame play experience relates to the way the user interacts with the gameand the game playing system, user input details tend to be important tothe success and marketability of home video game play systems.

One aspect of the video game user interface relates to how the usercontrols the position of one or more objects on the display. Much workhas been done on this user interface aspect in the past. For example,the first Magnavox Odyssey home video game systems provided detachablehandheld controllers with knobs that allowed the game player to controlthe horizontal and vertical positioning of objects on the screen. Pong®,another early home video game system, had a very simple user interfaceproviding controls the players manipulated to control the positioning ofpaddles on the screen. Nintendo's Game and Watch® early handheld videogame systems used a “cross-switch” as described in Nintendo's U.S. Pat.No. 4,687,200 to control the position of objects on the screen. Thesewere relatively simple yet effective user interfaces.

In recent years, video game system handheld controllers have tended tobecome increasingly more complicated and more capable. Video gameplatforms offered by Nintendo and others have provided joysticks,cross-switches or other user-manipulable controls as a means forallowing the user to control game play in a variety of simple andsophisticated ways. Many handheld controllers provide multiple joysticksas well an array of trigger buttons, additional control buttons, memoryports, and other features. Rumble or vibration effects are now common,as are wireless capabilities. Home video game manufacturers supply avariety of user input devices, and game accessory manufacturers oftenprovide an even wider array of input device options. For example, somein the past have also tried to develop a video game handheld controllerthat senses the orientation of the handheld controller itself to controlobject position on the display. See U.S. Pat. No. 5,059,958 assigned tothe present assignee.

One challenge that some have confronted in the past relates tocross-platform video game play. Generally, most video game systemmanufacturers differentiate new gaming systems from other or previousones by providing unique user interface features including for examplehandheld controller configurations. Video games for play on differenthome video game platforms may therefore use different handheldcontroller configurations. While it may be possible in some cases to“remap” the user controls from one interface configuration to another soa game for one platform can be controlled using a different inputcontrol interface, such remapping may be less than optimal and/or changethe game play experience in significant ways. For example, playing agame using a four-active-position cross-switch to control the movementof the main character on the screen may be quite a different experiencefor the user as compared with using an analog or digital joystickoffering many different directional positions.

Furthermore, most video game platforms in the past have provided asingle basic user interface that is used for all games playable on theplatform. Even though different video games may provide quite differentgame play, video game developers have become skilled at using the commonset of user input controls provided by the platform to control variousdifferent games. For example, most games developed to run on theNintendo GameCube home video game system make use of the same handheldcontroller inputs comprising two joysticks, trigger switches andadditional miscellaneous controls. Some games allocate differentcontrols to different functions. For example, in one game, the left-handjoystick might navigate a 2D map view of a battlefield whereas inanother game that same control might be used to allow the user to adjustvirtual camera position or direction within a three-dimensional world.

The technology herein advances home video game user interfaces in waysnot previously envisioned, to provide a more flexible and satisfyinguser experience across an ever increasing and divergent range of videogames and other applications.

One illustrative non-limiting exemplary aspect of the technology hereinprovides for positioning video game objects on the screen in response tothe position of a handheld controller relative to the display. Ratherthan moving a joystick or cross-switch, the user simply moves the entirehandheld controller. The motion of the controller is sensed and used tocontrol the position of objects or other parameters in connection withvideo game play.

Another exemplary non-limiting illustrative aspect of the technologyherein provides a handheld controller with a modular design. The basiccontroller functionality including wireless connectivity, vibrationgeneration, position sensing, orientation sensing and other features areprovided within a core or basic handheld controller unit. This core unitcan control many or most videogame input functions and play most games.However, for enhanced input functionality, the core unit can be pluggedinto an expansion controller assembly providing additional controls,inputs and other functionality. As one example, the core unit can beplugged into a first accessory expansion unit providing touch pads whenit is desired to play videogames requiring touch pad input. The samecore unit can be plugged into a different expansion unit providingjoysticks and other input devices to play videogames designed forjoystick inputs. The same core controller can be plugged into a stilladditional expansion unit when the player wishes to interact with avideogame system using a simpler control interface providing across-switch and additional input buttons. In one exemplary illustrativenon-limiting implementation, some of the accessory units are designed tomimic earlier or different videogame platforms to allow the videogamesystem to match user interactivity experiences provided by such othersystems.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages will be better and morecompletely understood by referring to the following detailed descriptionof exemplary illustrative non-limiting implementations in conjunctionwith the drawings, of which:

FIG. 1 shows an exemplary illustrative videogame system being operatedin a typical home game playing environment;

FIG. 2 shows an exemplary illustrative non-limiting implementation of ahandheld videogame controller;

FIGS. 2A-2E show different views of the FIG. 2 implementation beinggrasped by the hand;

FIG. 2F shows exemplary two-handed operation;

FIGS. 3 and 3A show exemplary illustrative variations of the FIG. 2controller with a top plate removed;

FIG. 4 shows a bottom view of the FIG. 2 controller;

FIG. 5 shows a bottom view of the FIG. 2 controller with bottom coverremoved;

FIG. 6 shows a side and front perspective view of the exemplary FIG. 2controller;

FIG. 6A shows an additional exemplary view of the FIG. 2 controllerincluding a head pivot or tilt feature;

FIGS. 6B-6H show different views of an alternative exemplaryillustrative non-limiting handheld controller implementation;

FIGS. 7A and 7B show different views of the FIG. 2 controller when usedto detect position relative to light emitters;

FIGS. 8A, 8B, 8B-1, 8C and 8D show exemplary illustrative non-limitingexpansion controller units into which the FIG. 2 core unit may beremovably disposed and interconnected;

FIG. 9 shows an exemplary illustrative non-limiting block diagramimplementation of the FIG. 1 system;

FIG. 10 shows an overall block diagram of the FIG. 2 controller;

FIG. 11 is an exemplary illustrative non-limiting block diagram of anoverall system; and

FIGS. 12A-12C show exemplary illustrative non-limiting block diagrams ofdifferent expansion unit controller configurations.

DETAILED DESCRIPTION

Example Overall Exemplary Illustrative Non-Limiting System

FIG. 1 shows an illustrative, exemplary non-limiting implementation of avideo game system 100. System 100 includes a main unit 102 sometimesalso called a “console.” Main unit 102 executes applications includingvideo game software, and generates images for display on the display 104of a conventional home color television set or other display device 106.Main unit 102 also generates sound for reproduction by TV set 106.People 108 can interact with the video game play to control or affectthe images and the progression of the game or other application.

Main unit 102 in the exemplary illustrative non-limiting implementationcan be used to play a variety of different games including drivinggames, adventure games, flying games, fighting games, and almost anyother type of game one might think of. The video game software that mainunit 102 executes may be delivered on bulk storage devices such asoptical disks, semiconductor memory devices or the like, it may bedownloaded into the main unit over a network, or it may be provided tothe main unit in any other desired manner. Main unit 102 may also becapable of performing applications in addition to video games (e.g.,movie playback, email, web browsing, or any other application one canimagine). A security system built into main unit 102 may ensure thatonly authorized or authentic applications are executed.

FIG. 1 shows people (“video game players”) 108 a, 108 b interacting withmain unit 102 to play a video game. While two players 108 are shown, anynumber of players may interact with the main unit 102 at any given time.In the exemplary illustrative non-limiting implementation shown, eachvideo game player 108 holds and operates a wireless handheld controlunit (“controller”) 200. The players 108 operate these controllers 200to generate input signals. The controllers 200 communicate their inputsignals wirelessly to main unit 102. Such wireless communications can beby any convenient wireless method such as radio transmission, infrared,ultraviolet, ultrasonic or any other desired technique. Wirelessperipherals could include Bluetooth, 802.11 (WiFi), HiperLAN/1,HiperLAN/2, HomeRF, VWB, WiMax or other. In other implementations, cordsor cables could be used to connect controllers 200 to main unit 102.

In the exemplary illustrative non-limiting implementation of system 100shown, players 108 operate handheld controllers 200 in various ways toprovide input signals to main unit 102. For example, players 108 maydepress buttons or otherwise manipulate other controls on controllers200 to generate certain input signals. The effect of such controlmanipulations in the exemplary illustrative non-limiting implementationdepends, at least in part, on the particular software that main unit 102is executing. For example, depressing a certain button may provide a“start game” or “pause game” in some contexts, and may provide differentfunctions (e.g., “jump character”) in other contexts.

In the illustrative exemplary non-limiting implementation shown,controllers 200 have internal capabilities for detecting position and/ororientation. In the exemplary illustrative non-limiting implementation,players may change the orientation or position of controllers 200 togenerate input signals. Controllers 200 may sense position and/ororientation and report that information to main unit 102. Main unit 102may use that information to control or affect video game play or otherfunctionality.

In one exemplary illustrative non-limiting implementation, each handholdcontroller 200 may include an internal position, attitude or orientationsensor that can sense the position, attitude and/or orientation of thecontroller relative to the earth's gravitational force. Such a sensormay for example comprise a 3-axis accelerometer that can senseorientation (or changes in orientation) of the controller 200 relativeto the direction of earth's gravitational pull. The output of such asensor may be reported to main unit 102 and used for example to controlmotion of a character displayed on display 104.

In addition, the exemplary illustrative non-limiting implementation ofsystem 100 shown in FIG. 1 includes wireless emitters 110 a, 110 b.These wireless emitters 110 may be placed on each side of display 104 inalignment with the edges of the screen. The wireless emitters 110 mayfor example each comprise one or more light emitting diodes or otherdevices 112 that emit infrared or other electromagnetic or otherradiation.

In one exemplary illustrative non-limiting implementation, the energythat emitters 110 emit has a wavelength or other characteristic thatallows the radiation to be readily distinguished from ambient radiation.In the exemplary illustrative non-limiting implementation, handheldcontrollers 200 each detect the radiation emitted by emitters 110 andgenerate signals indicative of the controller's relative position and/ormovement. Multiple controllers 200 can sense the same emitted radiationand generate different signals depending on the position or movement ofthat particular controller. Controllers 200 report the relative positionand/or movement signal to main unit 102. Main unit 102 may take anyappropriate action in response to such signals such as, for example,moving, rotating or otherwise changing a game character or other objector background on the display 104, scrolling a screen, selecting adifferent game function, or taking other actions.

In the exemplary illustrative implementation shown, the emitters 110 areadded or retro-fitted onto a conventional color television set 106 byfor example using an adhesive to attach the emitters onto the tophousing of the television set on the extreme left and right of thehousing in alignment with the edges of display 104. In this exemplaryillustrative non-limiting implementation, emitters 110 can be connectedto main unit 102 by cables or wires run behind the television set 106.In other implementations, emitters 110 could be built-in to televisionset 106 or mounted separately (e.g., on a set top box or otherwise). Instill other implementations, emitters 110 could possibly be replacedwith small reflective surfaces attached by adhesive to corners ofdisplay 104, and controllers 200 could emit electromagnetic radiationand receive reflections from the reflective surfaces (e.g., whose angleof incidence is equal to angle of reflectance). In still otherimplementations, controllers 200 could emit electromagnetic radiationsand units 110 could include sensors that sense the emitted radiation.Other implementations are possible.

Example Illustrative Non-Limiting Handheld Controller Design

FIG. 2 shows a perspective view of an exemplary illustrativenon-limiting implementation of controller 200. Controller 200 provides ahousing 202 that is graspable by one hand (see FIGS. 2A, 2B, 2C).Controller 200 in the exemplary illustrative non-limiting implementationis compact and has a solid rugged feel to it. It can be dropped onto ahard surface without breaking. Portions of its housing 202 are curved tofit comfortably into the hand (see FIGS. 2A, 2B, 2C).

As shown in FIG. 2A, the thumb can be positioned to operate controls ona top control surface 204 while the fingers are comfortably wrappedaround the controller's bottom surface 203. The digits of the hand(including the thumb) can operate the different controls arrayed on atop control surface 204 and elsewhere on the controller without fatigueand without wasted or undue motion. The controller 200 is small andlightweight enough to be comfortably held and supported for long periodsof time without fatigue. Controller 200 is dimensioned to exactly andcomfortably fit the average hand—not too small, not too big. Thecontrols are arranged such that the controller 200 can be operatedequally easily by the right hand or the left hand.

The controller housing 202 provides a top control surface 204 providingan array of controls depressible with the digits (fingers and/or thumbs)of the user's hand. In one illustrative non-limiting implementation, theuser may operate a direction switch 206 with a thumb or forefinger toindicate a direction in two dimensions. In the illustrative non-limitingexemplary implementation shown, the directional switch 206 may comprisea switch surface 208 that can be rocked in different directions toprovide different direction signals. The simplest form of such adirectional switch 206 may comprise a so-called “cross switch” (a switchin the shape of a cross) that can be rocked in four different directionsto provide four different, mutually exclusive direction signals (i.e.,up, down, left, right). A somewhat more flexible form of a directionalswitch 208 may comprise a circular switch surface 208 that can be rockedin any of a number of different directions to provide correspondingdifferent control signals indicating for example twelve, sixteen or moredifferent directions. Other directional switch configurations could beused to provide a much higher number of directional inputs approaching,equaling or exceeding the number of signals from an analog or digitaljoystick. A touch or “joy” pad, a pointing stick, a trackball, or otherinput device could be used instead of or in addition to a switch. If ajoypad were used, it could likely be operated in a direction-indicatingmode as opposed to a “drag displacement” mode. Other arrangements couldinclude touch sensitive display(s) or other types of displays.

Top control surface 204 in the exemplary illustrative non-limitingimplementation also provides a pair of thumb-operated control switches210 a, 210 b. These control switches 210 a, 210 b can be oriented asshown, or they could each be rotated say 45 degrees so as to beangularly displayed from one another in order to expose more surfacearea to a thumb positioned to operate either control switches 210 ordirectional switch 206. Control switches 210 a, 210 b could be used toactuate a variety of game or other functions including for example“start” and “select” functions.

Top control surface 204 may also provide an additional push button 214operated by the thumb for other functionality selection. A slide switch216 on the side of housing 202 may be operated to provide on/off orother functionality. Depending on requirements, a slide switch 216 couldbe located on either or both side surfaces of the exemplary controller200.

Top control surface 204 in the exemplary illustrative non-limitingimplementation further provides two additional controls 212 a, 212 bthat may comprise indicator lamps or lights. Alternatively, suchcontrols 212 could comprise additional operable controls such as pushbutton switches, so-called “pointing stick” type input devices, or otherinput devices. These controls 212 may be relatively dormant or littleused (while not being subject to accidental operation) when thecontroller 200 is operated in the hand positions shown in FIGS. 2A, 2B,2C, 2D, 2E, 2F. However, another way of using controller 200 is to holdthe controller in one hand (or place it on a flat surface such a table)and operate its controls with the forefinger and other fingers of theother hand. In such an alternate operating mode, the forefinger could beused to operate controls 212 a, 212 b if they are activatable inputdevices as opposed to indicators. FIG. 2D for example shows that in oneexemplary illustrative implementation, the user may move his or herthumb forward or backward to access different controls. FIG. 2D showsthe ability to move the thumb side to side to provide different controlactuations. FIG. 2F shows an exemplary illustrative non-limitingimplementation whereby the user can hold the handheld controller in bothhands and operate it with both left thumb and right thumbsimultaneously.

FIG. 3 shows an exploded view of controller 200 with a top plate 204removed to reveal a printed circuit board 220. Metallic pathways (notshown) and associated solder or other electrical interconnections may beused to electrically interconnect components via PC board 220. Variouscomponents including integrated circuit chips 222 (e.g., a wireless RF“Bluetooth” or other communications device, an accelerometer and othercomponents) may be mounted to the printed circuit board 220. The printedcircuit board 220 may also serve as a mounting surface for thedirectional switch 206, controls 210, 212, etc. The printed circuitboard 220 in one exemplary illustrative non-limiting implementationprovides a rugged fiberglass structure used to both mount andelectrically interconnect components of controller 200. The same ordifferent printed circuit board 220 may provide an edge or otherconnector 224 for use in electrically connecting controller 200 to otherdevices (to be described below). FIG. 3A shows a different exemplaryillustrative non-limiting implementation with a different exemplarynon-limiting control layout. Further configurations are also possible.

FIG. 4 shows a bottom view of an exemplary illustrative non-limitingimplementation of controller 200. The bottom view reveals an accessplate 230 for installing one or more small conventionalremovable/replaceable battery cells (see FIG. 5). FIG. 4 also shows anadditional “trigger” type switch 232 operable by the forefinger when thecontroller is held in the hand (see FIG. 2A, 2C). “Trigger” switch 232may for example sense pressure to provide a variable input signal thatdepends on how much pressure the user's forefinger is exerting on theswitch. Such a variable-pressure “trigger” switch 232 can be used in avideo game to fire weapons, control the speed of a vehicle in a drivingor space game, or provide other functionality.

In the exemplary illustrative non-limiting exemplary implementationshown, the trigger switch 232 is disposed on an angular surface 234 ofthe bottom surface 240 of controller 200 within a V-shaped depression236 located near the front distal end 238. This V-shaped depression 236is dimensioned to comfortably provide a resting and grasping slot forthe forefinger (see FIG. 2C) which may be slightly rotated and pulledtoward the user between a resting position (see FIG. 2C) and anactuation position (see FIG. 2A). With the middle, ring and pinkiefingers wrapped around and grasping the curved center 240 c and rear 240r portions of the controller's bottom surface 203 and the forefingercomfortably engaged within the v-shaped depression 236, the user feelsquite comfortable holding and operating controller 200 with one hand andpositioning and aiming it precisely in desired directions.

FIG. 5 shows an exploded view of controller 200 with the lower housingportion 240 removed to expose internal components such as removablyreplaceable batteries 250 and associated holders/contacts 252, andtrigger switch 232. While two batteries 250 are shown in FIG. 5, anynumber of batteries (e.g., one, three, etc.) can be used depending onweight, power and other requirements. Note that to replace batteries250, the user would not usually remove the lower housing 240 but ratherwould simply remove the access plate 230. In other configurations, thecontroller 200 might be rechargeable and batteries 250 could be of thenickel-cadmium or other type that do not require routine replacement. Insuch exemplary configuration, the controller 200 could be placed into acharging station to recharge the batteries 250 instead of expecting theuser to replace the batteries. While FIG. 5 shows a separate edgeconnector 224, it is possible that the edge connector could be formed bya distal edge of the printed circuit board 220.

FIGS. 6B-6H show an additional exemplary non-limiting illustrativeimplementation of a handheld controller with a different controlconfiguration. A power button 1002 may be used to activate power on themain unit 102. A control pad 206 provides directional input. An A button1004 can be operated by the thumb instead of the control pad 206 toprovide a momentary on-off control (e.g., to make a character jump,etc.). Select and start buttons 1006, 1008 may be provided for exampleto start game play, select menu options, etc. A menu button 1010 (whichmay be recessed to avoid accidental depression) may be provided todisplay or select menu/home functions. X and Y buttons may be used toprovide additional directional or other control. Light emitting diodesor other indicators 1016 a-d may be used to indicate various states ofoperation (e.g., for example to designate which controller number in amulti-controller environment the current controller is assigned). Aconnector 1018 is provided to connect the controller to externaldevices. FIG. 6C shows an underneath side perspective view, FIG. 6Dshows a top plan view, FIG. 6E shows a side plan view, FIG. 6F shows abottom plan view, FIG. 6G shows a front plan view, and FIG. 6H shows arear plan view.

Example Illustrative Non-Limiting Optical Pointing Device MotionDetection

FIG. 6 shows a front perspective view of controller 200 illustrating anadditional sensing component 260 also shown in FIG. 5. Sensor 260 in theexemplary illustrative non-limiting implementation is disposed on the“nose” or front surface 262 of controller 200 so that it points forward,looking down a pointing axis P. The direction of pointing axis P changesas the user changes the orientation of controller 200. It is possible toprovide a pivot mechanism (see FIG. 6A) to allow the user to pivot thenose portion up and down to provide better ergonomics (e.g., the usercould be sitting on the floor below the level of the emitters 112 andstill be able to point directly forward, with the sensor 260 axis Pbeing aimed upwardly).

Sensing component 260 in the exemplary illustrative non-limitingimplementation comprises an infrared-sensitive CCD type image sensor.Sensor 260 may comprise a one-dimensional line sensor or it couldcomprise a 2D sensor such as for example a low resolution monochrome CCDor other camera. Motion tracking sensor 260 may include a lens and aclosely coupled digital signal processor to process incoming images andreduce the amount of information that needs to be conveyed to main unit102. In one exemplary non-limiting implementation, motion trackingsensor 260 may include a 128 pixel by 96 pixel relatively low resolutionmonochrome camera, a digital signal processor and a focusing lens. Morethan one such sensor could be used if desired.

In the exemplary illustrative non-limiting implementation, sensor 260gives controller 200 optical pointing capabilities. For example,movement of the controller 200 can be detected (e.g., by the controlleritself) and used to control what is being displayed on display 104. Suchcontrol could include for example scrolling of the screen, rotation orother reorientation of display objects in response torotation/reorientation of controller 200, and other responsiveinteractive displays. Such control may provide a better moment arm ascompared to a joystick.

In the exemplary illustrative non-limiting implementation, sensor 260 isdesigned and configured to sense the emitters 110 shown in FIG. 1. FIGS.7A, 7B show that sensor 260 has a certain well defined field of view(FOV) symmetrical with the sensor pointing axis P. For example, thesensor 260 may have a field of view of about 20.5 degrees on each orevery side of pointing axis P (this particular field of view angle is adesign choice; other choices are possible in other configurations). Suchwell defined field of view provides an acute triangularly shaped (orcone-shaped for 2D sensor configurations) viewing area that sensor 260can “see”—with the base of the triangle increasing in length as distancefrom the controller 200 increases. Sensor 260 also has a well definedsensitivity such that it can only “see” IR emissions above a certainrange of intensity. Emitters 112 are designed in the exemplaryillustrative non-limiting to provide sufficient output power and beamspreading consistent with the sensitivity of sensor 260 such that sensorcan “see” the emitters at ranges consistent with how video game playersarrange themselves in a room relative to a television set 106 (takinginto account that a player may sometimes sit close to the televisionwhen playing by himself, that players may be sitting on the floor,standing, sitting on chairs or couches or other furniture, etc.).

In more detail, FIG. 7A shows that in the exemplary illustrativenon-limiting implementation, the overall field of view of sensor 260 iswider than the typical separation of emitters 112 and is also wider thanbeam width of each emitter 112. In one exemplary illustrativenon-limiting implementation, the ratio of the beam spreading angle(e.g., 34 degrees) of the beams emitted by emitters 112 to the field ofview (e.g., 41 degrees) of sensor 260 may be approximately 0.82 (otherratios are possible). Plural emitters 112 can be used at each emissionpoint to provide a wider beam (horizontal field of view) than mightotherwise be available from only a single emitter, or a lens or otheroptics can be used to achieve desired beam width.

At an average distance from controller 200 to television set 106 andassociated emitters 112 and assuming a maximum television screen size(and thus a maximum physical separation between the emitters), such aratio may maximize the displacement of two radiation “dots” or pointsappearing on the CCD sensor array 270 that sensor 260 comprises.Referring to FIG. 7A for example, when the central axis of sensor 260 isdirected centrally between displaced emitters 112 (note that in oneexemplary illustrative non-limiting implementation, the emitters aredisposed on either side of the television display and are thereforerelatively far apart relative to the resolution of the image beinggenerated), the CCD array 270 that sensor 260 defines will registermaximal illumination at two points near the ends of the sensor array.This provides a higher degree of resolution when the sensor 260'scentral axis P is displaced relative to the center of separation of theemitters 112 (see FIG. 7B) even when using a relatively low resolutionCCD imaging array (e.g., a 128-cell long sensor array). Note that whilea linear array 270 is illustrated in FIGS. 7A, 7B for sake ofconvenience, a rectangular array could be used instead.

In the illustrative, exemplary non-limiting implementation shown, it isunnecessary to modulate or synchronize emitters 112 in the exemplaryillustrative non-limiting implementation, although it may be desirableto power down the emitters when not in use to conserve power usage. Inother arrangements, however, synchronous detection, modulation and othertechniques could be used.

The exemplary illustrative non-limiting implementation of controller 200and/or main unit 102 includes software or hardware functionality todetermine the position of controller 200 relative to emitters 112, inresponse to the illumination maxima sensed by sensor 260. In one exampleillustrative non-limiting implementation, controller 200 includes anon-board processor coupled to the sensor 260 that interprets thecurrently detected illumination pattern, correlates it with previoussensed illumination patterns, and derives a current position. In anotherexample illustrative non-limiting implementation, controller 200 maysimply report the sensed pattern to main unit 102 which then performsthe needed processing to detect motion of controller 200. The sensorcould be affixed to the human operating the system to provide additionalcontrol.

Since it may not be desirable to require end users of system 100 tomeasure and program in the precise distance between the emitters 112 andsince television sets vary in dimension from small screens to very largescreens, controller 200 does not attempt to calculate or derive exactpositional or distance information. Rather, controller 200 may determinemovement changes in relative position or distance by analyzing changesin the illumination pattern “seen” by CCD array 270.

It may be possible to ask the user to initially point the controller 200at the center of the television screen 104 and press a button, so as toestablish a calibration point (e.g., see FIG. 7A)—or the game player maybe encouraged to point to the center of the screen by displaying anobject at the center of the screen and asking the user to “aim” at theobject and depress the trigger switch. Alternatively, to maximize userfriendliness, the system can be self-calibrating or require nocalibration at all.

Differences in the illumination pattern that CCD array 270 observesrelative to previously sensed patterns (see e.g., FIG. 7B) can be usedto determine or estimate movement (change in position) relative toprevious position in three dimensions. Even though the CCD array 270illumination shown in the FIG. 7B scenario is ambiguous (it could beobtained by aiming directly at emitter 112 a or at emitter 112 b),recording and analyzing illumination patterns on a relatively frequentperiodic or other basis (e.g., 200 times per second) allows thecontroller to continually keep track of where it is relative to theemitters 112 and previous controller positions. The distance between theillumination points of emitters 112 and CCD array 270 can be used toestimate relative distance from the emitters. Generally, game playerscan be assumed to be standing directly in front of the television setand perpendicular to the plane of display 106. However, scenarios inwhich controller 200 is aimed “off axis” such that its central axis Pintersects the plane of emitters 112 at an angle other thanperpendicular can also be detected by determining the decreasedseparation of the two maximum illumination points on the CCD array 270relative to an earlier detected separation. Care must be taken howeversince changes in separation can be attributed to changed distance fromthe emitters 112 as opposed to off-axis. Simpler mathematics can be usedfor the motion and relative position detection if one assumes that theplayer is aiming the sensor axis P directly at the display 104 so theaxis perpendicularly intersects the plane of the display.

Software algorithms of conventional design can ascertain position ofcontroller 200 relative to emitters 112 and to each logical or actualedge of the display screen 104. If desired, controller 200 may furtherinclude an internal conventional 3-axis accelerometer that detects theearth's gravitational forces in three dimensions and may thus be used asan inclinometer. Such inclination (orientation) information in threeaxis can be used to provide further inputs to the relativeposition-detecting algorithm, to provide rough (x, y, z) positioninformation in three dimensions. Such relative position information (orsignals from which it can be derived) can be wirelessly communicated tomain unit 102 and used to control the position of displayed objects onthe screen.

Example Modular Control Interface Controller Expansion

FIGS. 8A-8D illustrate an additional feature of the exemplaryillustrative non-limiting implementation of controller 200. Inaccordance with this additional feature, the controller 200 may be usedas the “core” of a modular, larger handheld controller unit byconnecting the controller 200 to an additional expansion unit 300. Corecontroller 200 may “ride piggyback” on an expansion unit 300 to easilyand flexibly provide additional control interface functionality that canbe changed by simply unplugging the controller from one expansion unitan plugging it in to another expansion unit.

FIG. 8A shows one exemplary illustrative non-limiting such additionalexpansion unit 300 including a housing 302 having a control surface 304providing an array of additional controls including for example ajoystick 306, a cross-switch 308 and various push-button controls 310.Expansion unit 300 includes a depression such that when the rear portionof controller 200 is inserted into the depression, the resultingcombined unit provides an overall planar T-shaped control surface thatcombines the expansion unit 300 control surface with the controller 200control surface in a flush and continuous manner. In such case, the usermay grasp the expansion unit 300 with two hands and may operate thecontrols of controller 200 (see FIG. 8B-1) or controls on the expansionunit 300. Expansion unit 300 thus effectively converts the controller200 designed to be held in a single hand into a two-handed controllerwhile also supplying additional controls.

FIG. 8B shows a further expansion unit 300′ having a somewhat differentcontrol configuration. FIGS. 8C and 8D show additional non-limitingillustrative example expansion units.

As shown in FIG. 8B-1, expansion units 300 may provide all of thecontrols that the user would operate to control a video game whencontroller 200 is plugged into the additional unit. This provides a highdegree of flexibility, since any number of additional units 300 of anydesired configuration can be provided. Such additional units 300 can bemanufactured relatively inexpensively since they can rely on controller200 for power, processing, wireless communications and all other corefunctions. In the exemplary illustrative non-limiting implementation,controller edge connector 224 exposes sufficient connections and asufficiently flexible interface such that an expansion unit 300 ofvirtually any desirable description can be compatibly used.

One possible motivation for manufacturing expansion units 300 is toprovide control interface compatibility with other video game platformsincluding for example legacy platforms such as the NintendoEntertainment System, the Super Nintendo Entertainment System, theNintendo 64, the Nintendo GameCube System, and the Nintendo Game Boy,Game Boy Advance and Nintendo DS systems. An expansion unit 300providing a control interface similar or identical to for the examplethe Super Nintendo Entertainment System could be made available forplaying Super Nintendo Entertainment System games on system 100. Thiswould eliminate the desire to reprogram or rework Super NintendoEntertainment System games for use with the newer or different interfaceprovided by controller 200.

Another possible, more general motivation for additional expansion units300 is to provide customized control interfaces for particular games orother applications. For example, it would be possible to develop a unit300 with a steering wheel for driving games, a unit with a keyboard fortext entry applications, a unit with one or multiple touch pads fortouch screen style games, etc. Any desired control configuration ispossible and can be flexibly accommodated.

Still another possible application would be to use expansion units 300to give different players of a multi-player game different capabilities.For example, one game player might use controller 200 “as is” withoutany expansion, another game player might use the expansion configurationshown in FIG. 12A, yet another game player might use the expansionconfiguration shown in FIG. 12B, etc. One could imagine a militarybattle game for example in which game players playing the role of tankdrivers use an expansion unit that resembles the controls of a tank,game players playing the role of artillerymen use an expansion unit thatresembles controls of heavy artillery, and a game player playing therole of a commanding general uses an expansion unit that provides moregeneral controls to locate infantry, artillery and tanks on the field.

Example Illustrative Non-Limiting Block Diagrams

FIG. 9 shows a block diagram of an exemplary illustrative implementationof system 100. As described above, system 100 includes a main unit 102and one or several controllers 200 a, 200 b, 200 c, etc. Each controller200 may be connected to any of additional expansion units 300 or may beused by itself, depending on the application. Additional wirelessperipherals to system 100 may include a headset unit 180 for voice chatand other applications, a keyboard unit 182, a mouse or other pointingdevice 184, and other peripheral input and/or output units.

FIG. 10 is a block diagram of an exemplary illustrative non-limitingimplementation of controller 200. In the example shown, controller 200may comprise a wireless connectivity chip 280 that communicatesbidirectionally with main unit 102 via a pattern antenna 278. Wirelesscommunications chip 280 may be based on the Bluetooth standard butcustomized to provide low latency. In the example shown here, most orall processing is performed by the main unit 102, and controller 200acts more like a telemetry device to relay sensed information back tothe main unit 102. Such sensed inputs may include a motion trackingsensor 260, an accelerometer 290, and various buttons 206, 210, etc. asdescribed above. Output devices included with or within controller 200may include a vibrational transducer 292 and various indicators 294.

FIG. 11 shows an overall exemplary illustrative non-limiting systemblock diagram showing a portion of main unit 102 that communicates withcontroller 200. Such exemplary illustrative non-limiting main unit 102portion may include for example a wireless controller 1000, a ROM/RealTime Clock 1002, an idle mode indicator 1004, a processor 1006 andvarious power supplies. Link buttons may be provided on each side of thecommunications link to provide manual input forsynchronization/training/searching.

FIGS. 12A, 12B and 12C show different exemplary block diagramconfigurations for different expansion units 300. The FIG. 12A exampleincludes dual touch pads 1200 and a joystick 1202 for touch screencompatible gaming; the FIG. 12B example includes two joysticks 1202 andother controls for games requiring two different joysticks (e.g.,Nintendo GameCube legacy games); and the FIG. 12C example includes across-switch 1204 and other controls for more limited user interfacetype games (e.g., Nintendo Entertainment System legacy games).

Each expansion unit may be programmed with a 4-bit or other length“type” ID to permit controller 200 to detect which type of expansionunit is being used. Main unit 102 can adapt user interactivity based atleast in part on the “type” ID.

While the technology herein has been described in connection withexemplary illustrative non-limiting implementations, the invention isnot to be limited by the disclosure. The invention is intended to bedefined by the claims and to cover all corresponding and equivalentarrangements whether or not specifically disclosed herein.

We claim:
 1. A wireless handheld remote controller configured to be heldin one hand, comprising: a housing including an upper surface and alower surface; at least one digit operable detector disposed on theupper surface; at least one depressible trigger disposed on said lowersurface; an inertial sensor mounted in the housing; a two dimensionalradiation detector; a processor that processes an output of theradiation detector and determines an illumination pattern; a wirelesstransceiver that transmits information based on signals generated by theinertial sensor and the processor; and an output device operativelycoupled to the transceiver.
 2. The controller of claim 1 wherein theradiation detector is disposed, at least in part, at a front portion ofthe housing.
 3. The controller of claim 1, wherein the radiationdetector comprises a two dimensional camera.
 4. The controller of claim1, wherein the radiation detector comprises: a two dimensional radiationsensor array; and an infrared filter that is mounted on the housing infront of the two dimensional radiation sensor array such that onlyinfrared light passing through the filter is received by the radiationsensor array.
 5. The controller of claim 1, wherein the radiationdetector generates frames of two dimensional image data, and wherein theprocessor determines an illumination pattern for each frame of imagedata.
 6. The controller of claim 5, wherein each illumination patterncomprises X and Y coordinates for illuminated objects appearing within aframe of image data.
 7. The controller of claim 5, wherein eachillumination pattern comprises X and Y coordinates for illuminatedobjects appearing within a frame of image data that have an intensitythat rises above a predetermined threshold value.
 8. The controller ofclaim 5, wherein each illumination pattern comprises X and Y coordinatesfor illuminated objects appearing within a frame of image data that emitinfrared radiation having an intensity that rises above a predeterminedthreshold value.
 9. The controller of claim 5, wherein the wirelesstransceiver transmits information regarding the illumination patternsfor frames of image data.
 10. The controller of claim 9, wherein theinertial sensor comprises an accelerometer.
 11. The controller of claim10, wherein the accelerometer is a three axis accelerometer that senseslinear acceleration in each of three mutually perpendicular axes, andwherein the inertial sensor outputs three linear acceleration valuescorresponding to the three mutually perpendicular axes multiple timesevery second.
 12. The controller of claim 11, wherein the wirelesstransceiver also transmits a set of the three acceleration valuesmultiple times every second.
 13. The controller of claim 1, wherein theinertial sensor comprises an accelerometer.
 14. The controller of claim13, wherein the accelerometer is a three axis accelerometer that senseslinear acceleration in each of three mutually perpendicular axes, andwherein the inertial sensor outputs three linear acceleration valuescorresponding to the three mutually perpendicular axes multiple timesevery second.
 15. The controller of claim 14, wherein the wirelesstransceiver transmits a set of the three acceleration values multipletimes every second.
 16. The controller of claim 1, wherein the outputdevice comprises a speaker, and wherein the speaker outputs sounds basedon a signal received by the wireless transceiver.
 17. The controller ofclaim 1, wherein the output device comprises a vibration module thatcauses the housing to vibrate based on a signal received by the wirelesstransceiver.
 18. The controller of claim 1, wherein the output devicecomprises at least one indicator light that is selectively illuminatedbased on a signal received by the wireless transceiver.
 19. Thecontroller of claim 1, wherein the output device comprises an array ofindicator lights that are selectively illuminated based on a signalreceived by the wireless transceiver.
 20. The controller of claim 1,wherein the at least one digit operable detector comprises at least onedepressible button disposed on the upper surface of the housing.